Search for: All records

Large pretrained transformer models have revolutionized modern AI applications with their state-of-the-art performance in natural language processing (NLP). However, their substantial parameter count poses challenges for real-world deployment. To address this, researchers often reduce model size by pruning parameters based on their magnitude or sensitivity. Previous research has demonstrated the limitations of magnitude pruning, especially in the context of transfer learning for modern NLP tasks. In this paper, we introduce a new magnitude-based pruning algorithm called mixture Gaussian prior pruning (MGPP), which employs a mixture Gaussian prior for regularization. MGPP prunes non-expressive weights under the guidance of the mixture Gaussian prior, aiming to retain the model’s expressive capability. Extensive evaluations across various NLP tasks, including natural language understanding, question answering, and natural language generation, demonstrate the superiority of MGPP over existing pruning methods, particularly in high sparsity settings. Additionally, we provide a theoretical justification for the consistency of the sparse transformer, shedding light on the effectiveness of the proposed pruning method. 

Abstract We report a metal–organic framework (MOF) with a rare two‐dimensional (2D) secondary building unit (SBU). The SBU comprises mixed‐valent Fe²⁺and Fe³⁺metal ions bridged by oxygen atoms pertaining to the polytopic ligand 3,3′,4,4′,5,5′‐hexahydroxybiphenyl, which also define the iron‐oxide 2D layers. Overall, the anionic framework exhibits rare topology and evidences strong electronic communication between the mixed‐valence iron sites. These results highlight the importance of dimensionality control of MOF SBUs for discovering new topologies in reticular chemistry, and especially for improving electronic communication within the MOF skeleton.